Repository navigation
ci(eval): harness sanity gate + baseline scaffold - #67
Merged
Merged
Conversation
Phase 3 (BugHunter). The full ablation eval (run_eval*.py — skills-on vs skills-off on a self-grading target) needs Docker + Burp + an authed claude CLI, so it runs locally, not in CI. But the harness around it can silently rot. This adds the CI-able half plus a place to record the real numbers: - .github/workflows/eval-harness.yml — on eval/ changes: py_compile every eval/*.py, run the oracle parser self-test, run the new harness sanity test. Network-free; contents: read. - eval/test_harness.py — asserts (1) every eval/*.json config parses with the keys the runners expect, (2) oracle_portswigger classifies the lab-status widget correctly, (3) fp_app.py starts and serves every path in fp_cases.json (keeps the FP-trap app and its ground-truth cases in sync). Stdlib, localhost. - eval/BASELINE.md — the local run procedure + a results table to fill in (skills-on/off solve-rate delta) and commit after a real run. Building test_harness caught two bugs in my own probe (not the app): appending a canary to an exact-match route corrupted it, and urllib auto-followed the open-redirect trap's 302 to a 404. Fixed (query-aware URL + no-follow opener); fp_app and fp_cases are in sync. The real skills-on/off baseline is produced by running the eval locally and committing BASELINE.md — CI can't run it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 (BugHunter) — the CI-able half of capability eval.
The full ablation eval (
run_eval*.py— skills-on vs skills-off on a self-grading target) needs Docker + Burp + an authedclaudeCLI, so it runs locally, not on a hosted runner. This ships the part CI can do — keeping the harness from rotting — plus a place to record the real numbers.What this adds
.github/workflows/eval-harness.yml— oneval/changes:py_compileeveryeval/*.py, run the oracle parser self-test, run the new harness sanity test. Network-free,contents: read.eval/test_harness.py— asserts (1) everyeval/*.jsonconfig parses with the keys the runners expect, (2)oracle_portswiggerclassifies the lab-status widget correctly, (3)fp_app.pystarts and serves every path infp_cases.json(keeps the false-positive trap app and its ground-truth cases in sync). Stdlib, localhost only.eval/BASELINE.md— the local run procedure + a results table to fill in (skills-on/off solve-rate delta) and commit after a real run.What's NOT here
The actual skills-on/off baseline numbers — those require your local Docker/Burp/authed-CLI setup. Run
eval/run_eval.py(Juice Shop v0) oreval/run_eval_ps_auto.py(PortSwigger v1), then fill in and commiteval/BASELINE.md.Note
Building
test_harness.pycaught two bugs in my own probe (not the app): appending a canary to an exact-match route corrupted it, and urllib auto-followed the open-redirect trap's 302 to a 404. Fixed (query-aware URL + no-follow opener);fp_appandfp_casesare confirmed in sync.Verified
py_compileall eval scripts · oracle self-test 4/4 · harness test PASS · leak-guard 0 matches.🤖 Generated with Claude Code